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The  Joint  Capabilities  Integration  and  Developnnent 
System  (JCIDS)  is  10  years  old  and  ripe  for  review.  A 
central  output  document  of  the  JCIDS  process  is  an  Initial 
Capabilities  Document  (ICD)  used  by  the  Department  of 
Defense  to  define  gaps  in  a  functional  capability  area 
and  define  new  capabilities  required.  The  research  team 
analyzed  10  years  of  ICDs  to  identify  methods  and  trends. 
The  team  found  that  several  methodologies  were  favored 
and  a  convergence  emerged  in  format  and  necessary 
content.  Additionally,  potential  shortfalls  in  current  best 
practices  of  interest  to  implementers  and  decision  makers 
are  identified.  Guidelines  and  best  practices  are  presented 
to  create  more  effective,  concise,  and  complete  ICDs. 
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It  may  come  as  a  surprise  to  many  acquisition  practitioners  that 
the  historically  unstable,  formal  written  procedures  and  processes 
that  embody  the  Defense  Acquisition  System  and  Joint  Capabilities 
Integration  and  Development  System  (JCIDS)  are  now  over  10  years 
old.  During  this  time,  the  Department  of  Defense  (DoD)  has  published 
significant  revisions  and  updates  to  the  JCIDS-related  documents, 
including  Department  of  Defense  Instruction  (DoDI)  5000.02  entitled. 
Operation  of  the  Defense  Acquisition  System  and  the  Joint  Capabilities 
Integration  and  Development  System  Manual  (DoD,  2013;  Joint 
Requirements  Oversight  Council  [JROC],  2012).  The  current  system’s 
longevity  may  be  partially  attributable  to  its  utilization  of  modern 
management  approaches,  further  enabled  by  a  slow  convergence  of  the 
Joint  Strategic  Planning  System  set  in  motion  by  the  Goldwater-Nichols 
Act  (Goldwater-Nichols,  1986).  With  its  focus  on  Joint  development 
and  deconfliction  of  capabilities,  JGIDS  uses  a  portfolio  management 
approach  and  streamlined  documentation  to  elevate  user  requirements 
relatively  quickly  and  vet  them  against  current  capabilities.  Further, 
its  emphasis  on  knowledge  management  ensures  that  all  stakeholders 
can  view  the  process  and  its  outcomes  as  the  key  documents  percolate 
through  the  JGIDS  process. 

Early  analysis  of  the  JGIDS  process  by  the  U.S.  Government 
Accountability  Office  (GAO,  2008)  identified  variable  product  quality. 
Attempts  were  made  at  creating  user’s  guides  to  improve  document 
quality  (JROG,  2012;  Joint  Ghiefs  of  Staff  [JGS],  2009);  however,  these 
documents  did  not  fully  address  the  analysis  techniques  contained 
therein.  As  a  key  component  of  process  quality,  the  ability  to  select,  use, 
and  report  an  appropriate  analysis  technique  is  an  item  of  interest  for 
authors,  stakeholders,  and  portfolio  managers.  Therefore,  this  effort 
reviewed  the  content,  tools,  and  methodologies  recorded  in  the  past  10 
years’  Initial  Gapabilities  Documents  (IGDs)  created  as  a  part  of  the 
JGIDS  process. 


Early  analysis  of  the  JCIDS  process  by  the  U.S. 
Government  Accountability  Office  (GAO,  2008) 
identified  variable  product  quality. 
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As  one  of  the  first  products  created  in  JCIDS,  ICDs  are  important 
because  they  validate  requirements  derived  through  an  analysis  of 
current  capabilities  and  capability  gaps.  Additionally,  they  are  signed 
by  senior  service  members  and  are  the  basis  for  program  acquisitions. 
Further,  due  to  their  recommended  brevity,  it  is  important  that  ICDs 
contain  the  correct  level  of  detail  to  identify  the  key  assumptions, 
limitations,  and  boundary  conditions  contained  or  referenced  in  their 
analyses.  A  lack  of  analytical  clarity  at  this  stage  may  lead  to  misdirected 
resources  further  in  the  process  (GAO,  2008). 

Of  particular  interest  were  the  methodologies  that  implementers 
and  decision  makers  were  choosing  to  use  in  developing  ICDs.  Through 
this  process,  it  was  possible  to  identify  a  series  of  best  practices  and 
guidelines  to  improve  ICD  quality,  and  thus  aid  in  the  evolution  of  JCIDS. 

Background 

The  JCIDS  process  was  created  as  a  response  to  a  2002  memo¬ 
randum  from  the  Secretary  of  Defense  to  the  Vice  Chairman  of  the 
Joint  Chiefs  of  Staff  to  study  alternative  ways  to  evaluate  requirements 
(JCIDS,  2014).  At  the  time  of  this  memorandum,  the  governing  document 
was  Chairman  of  the  Joint  Chiefs  of  Staff  Instruction  3170.01B  (CJCSI, 
2001)  and  was  titled  the  Requirements  Generation  System.  The  purpose 
of  JCIDS  was  to  streamline  and  standardize  the  methodology  to  identify 
and  describe  capabilities’  gaps  across  the  DoD,  and  to  engage  the  acqui¬ 
sition  community  early  in  the  process  while  improving  coordination 
between  departments  and  agencies. 

The  GAO’s  (2008)  report  indicated  that  “the  JCIDS  process  has  not 
yet  been  effective  in  identifying  and  prioritizing  warfighting  needs  from 
a  joint,  department-wide  perspective”  (GAO,  2008,  para.  1).  This  report 
outlined  the  shortfalls  and  gaps  in  the  JCIDS  process  in  its  5-year  life 
span,  furthering  the  redesign  of  the  process.  Additionally,  the  report 
outlined  several  recommendations  for  the  DoD,  including  developing 
a  more  analytical  approach  within  JCIDS  to  better  prioritize  and  bal¬ 
ance  capability  needs  as  well  as  allocating  the  appropriate  resources  for 
capabilities  development  planning. 

The  current  documentation  for  both  creating  and  implementing  ICDs 
are  the  Capabilities-Based  Assessment  (CBA)  User’s  Guide  and  the  JCIDS 
Manual.  These  documents  were  released  in  2009  and  2012  respectively 
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as  a  part  of  the  process  to  address  the  issues  found  by  the  2008  GAO 
report.  The  impact  of  these  documents  in  improvements  to  the  JCIDS 
process  has  yet  to  be  determined,  but  will  be  discussed  in  this  article. 

Focus  and  Methodology 

The  research  team  used  the  Knowledge  Management/Decision 
Support  (KM/DS)  system  to  examine  the  JCIDS  process.  The  KM/DS 
Web  site  is  the  repository  for  the  documents  created  through  or  as  a 
byproduct  of  the  JCIDS  process.  Included  in  this  study  are  ICDs,  Joint 
Capabilities  Documents  (JCDs),  Capability  Development  Documents, 
and  other  supporting  documents  that  are  a  part  of  this  process.  To  focus 
this  research,  the  team  specifically  studied  the  core  documents— ICDs 
and  JCDs— to  better  understand  what  kinds  of  methodologies  are  being 
implemented  by  the  various  Services  to  convey  the  gap  information 
under  study. 


Ultimately,  it  was  the  intention  of  the  research  team 
to  observe  and  report  on  best  practices  for  future 
ICD  writers. 


Of  those  entered  in  the  KM/DS  system,  over  1,000  ICDs  and  JCDs 
were  in  various  phases  of  the  JCIDS  process  covering  the  period  January 
1,  2002,  to  December  31,  2012.  The  team  decided  to  focus  on  only  those 
documents  that  were  considered  ‘Validated’  and  ‘Final,’  with  the  expec¬ 
tation  of  little  to  no  revision  remaining  for  these  documents  in  the  near 
future.  These  criteria  reduced  the  number  of  the  documents  under  review 
to  225  ICDs/JCDs.  The  team  of  four  researchers  split  the  ICDs/JCDs 
evenly  across  year  and  type  to  ensure  similar  exposure  to  the  complete 
population  available.  At  the  completion  of  the  review,  the  researchers 
met  and  discussed  commonalities  and  anomalies  found  in  documents 
of  interest,  and  in  the  population  in  general.  For  purposes  of  this  article, 
the  term  ICD  will  be  used  to  describe  both  the  ICDs  and  JCDs  unless 
specified  otherwise. 

The  team  formulated  an  initial  set  of  generally  accepted  methodologies 
for  a  baseline  to  identify,  categorize,  and  sort  the  currently  used  methodolo¬ 
gies  within  the  ICDs.  They  did  not  solely  consider  this  set  of  techniques,  but 
allowed  for  an  expansion  of  the  list  to  detect  emergent  techniques. 
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Additionally,  an  analysis  was  performed  on  key  metrics  and  areas  of 
interest  to  see  if  there  were  any  correlations  or  observations  that  could 
be  made  about  various  components  of  the  ICDs.  These  attributes  were 
chosen  as  they  were  key  areas  of  interest  or  sections  in  the  Capabilities- 
Based  Assessment  (CBA)  User’s  Guide  and  the  JCIDS  Manual.  By 
examining  these  attributes,  the  team  was  able  to  determine  to  what 
extent  past  ICDs  have  followed  current  guidance.  Some  of  the  compo¬ 
nents  considered  in  the  analysis  can  be  found  in  Table  1. 


TABLE  1.  ATTRIBUTES  FOR  ANALYSIS 


Attributes 

ACAT  Level 

DOTMLPF-P 

Analysis 

Measures  of 

Effectiveness 

Threshold 

Values  Defined 

Lead  FCB 

Formatting 

UJTL 

Traceability 

Objective 

Values  Defined 

Supporting 

FCBs 

Analysis 

Described 

Number  of  Gaps 

Number  of 

Pages 

Current 

Milestone 

Capabilities 

Defined 

Gap 

Prioritization 

Attributes 

Listed 

Note.  ACAT  =  Acquisition  Category;  DOTMLPF-P  =  Doctrine,  Organization,  Training, 
Materiei,  Leadership,  Personnel,  Facilities  -  Policy;  FCB  =  Functional  Capabilities  Board; 
UJTL  =  Universal  Joint  Task  List. 


Ultimately,  it  was  the  intention  of  the  research  team  to  observe  and 
report  on  best  practices  for  future  ICD  writers.  As  such,  we  focused 
on  finding  those  ICDs  that  best  embodied  the  intentions  found  in  the 
Capabilities-Based  Assessment  (CBA)  User’s  Guide  (JCS,  2009)  and  the 
JGIDS Manual  (JROC,  2012). 


Results 

The  team  examined  several  ICD  characteristics  that  are  presented 
in  the  JGIDS  Manual  and  were  expected  to  be  used  in  most  ICDs  (Figure 
1).  The  team  found  that  of  the  features  prescribed  by  the  JGIDS  Manual, 
many  were  not  present  in  the  majority  of  ICDs  reviewed.  Less  than  half 
of  the  ICDs  described  what  analysis  was  done  to  identify  capability  gaps. 
Over  90  percent  of  the  ICDs  reviewed  define  a  specific  capability  while 
some  ICDs  do  not  have  a  well-defined  end  state. 
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Nearly  half  of  the  ICDs  analyzed  defined  their  Measures  of 
Effectiveness  (MOE),  described  their  analysis,  prioritized  gaps  and  capa¬ 
bilities,  and  defined  minimum  values  for  required  capability  attributes. 
The  presence  of  these  characteristics  provides  additional  information  to 
the  reader  and  improves  the  fidelity  of  the  ICD;  their  absence  leaves  com¬ 
monly  questioned  areas  open  for  discussion.  The  2012  JCIDS  Manual 
requires  threshold  values,  but  description  of  the  analysis  has  been  left 
open  to  the  document  creator,  and  many  choose  not  to  describe  it.  In 
fact,  the  manual  states  a  preference  to  ’’avoid  unnecessary  rigor  and 
time-consuming  detail.”  Applying  and  documenting  some  level  of  rigor 
seems  necessary  and  useful  for  documenting  how  gaps  were  identified 
and  showing  how  the  capability  requirements  were  justified.  The  priori¬ 
tization  of  gaps  and  capabilities  helps  decision  makers  understand  those 
components  that  are  critical  when  resources  are  limited  to  address  the 
full  capability  gap,  but  allows  for  partial  capability  fulfillment  or  a  subset 
of  smaller  gaps  to  be  filled. 

The  inclusion  of  an  Analysis  of  Alternatives  (AoA)  is  an  interesting 
additional  piece  of  content  as  it  is  no  longer  part  of  the  Capabilities-Based 
Assessment  (CBA)  User’s  Guide,  and  is  done  in  subsequent  work  of  the 


FIGURE  1.  ICD  CONTENT  ANALYSIS 


Note.  AoA  =  Analysis  of  Alternatives;  MOEs  =  Measures  of  Effectiveness;  UJTL  = 
Universal  Joint  Task  List. 


721 


Defense ARJ,  July  2014,  Vol.  21  No.  3: 716-749 


Initial  Capabilities  Documents:  A  10-Year  Retrospective  of  Tools,  Methodologies,  and  Best  Practices 


JCIDS  process.  Nearly  one-third  of  all  ICDs  included  some  form  of  an 
AoA,  whether  in  the  form  of  a  brief  paragraph  or  a  full  documentation 
found  in  attachments  or  enclosures.  Most  documents  that  contained  a 
complete  AoA  were  from  the  first  5  years,  a  period  of  time  in  which  the 
content  of  ICDs  was  still  in  flux.  Including  an  AoA  would  presuppose  a 
preferred  materiel  solution— something  not  within  the  scope  of  docu¬ 
menting  a  capability  gap. 

Also,  less  than  25  percent  of  the  ICDs  surveyed  contained  objective 
values  for  the  capabilities  to  be  met.  While  it  has  become  more  common 
for  threshold  values  to  be  defined  for  capabilities,  objective  values  can 
only  be  seen  in  less  than  half  of  those  cases.  One  might  expect  to  see 
objective  values  used  more  frequently  to  quantify  desired  capabilities 
beyond  the  minimums.  Including  objective  values  is  expected  to  aid  the 
process  owner  in  determining  if  a  recommended  solution  is  able  to  meet 
the  objective  of  closing  the  specified  gap. 

FIGURE  2.  NUMBER  OF  FUNCTIONAL  CAPABILITIES  IN  ICDS 

ANALYZED 


Functional  Capabilities  Board 


SUPPORTING 

LEAD 

LEAD  CUMULATIVE 
SUPPORTING  CUMULATIVE 


Note.  C4  =  Command,  Control,  Communications,  and  Computers;  FCBs  =  Functional 
Capabilities  Boards. 
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Identifying  the  Functional  Capabilities  Boards  (FCBs)  to  which 
ICDs  were  assigned  provided  insight  as  to  what  types  of  capabili¬ 
ties  have  been  defined  and  what  priorities  have  been  dictated.  FCB 
and  associated  Joint  Capability  Area  (JCA)  categories  include 
Force  Support  (formerly  Force  Support  and  Building  Partnerships); 
Battlespace  Awareness;  Force  Application;  Logistics;  Command, 
Control,  Communications,  and  Computers  (C4)/Cyber  (formerly 
Net-Centric,  Command  and  Control,  and  C4/Cyber);  and  Protection. 
Previous  FCBs,  including  Special  Operations  and  Test,  are  listed  in 
Figure  2  under  “Other  Legacy  FCBs.” 


Identifying  the  Functional  Capabilities  Boards 
(FCBs)  to  which  ICDs  were  assigned  provided 
insight  as  to  what  types  of  capabilities  have  been 
defined  and  what  priorities  have  been  dictated. 


FIGURE  3.  AVERAGE  NUMBER  OF  PAGES  FOR  ICDs  IN 
CORRESPONDING  YEARS 


Average  Pages  by  Year 

40 
35 
30 
25 
20 
15 
10 
5 
0 

2003  2004  2005  2006  2007  2008  2009  2010  2011  2012 


AVG  PGS  =  27 
STD  DEV  =  7.58 
MEDIAN  =  22.39 


Note.  Avg  =  Average;  Dev  =  Deviation;  Std  =  Standard. 
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Each  ICD  is  assigned  a  lead  and  supporting  FCB.  Figure  2  shows 
ICDs  arranged  by  lead  FCB  with  Force  Application  being  the  most  promi¬ 
nent  lead  FCB.  The  prominence  of  Force  Application  over  Force  Support 
led  the  team  to  conclude  that  validated  ICDs  are  more  likely  to  focus  on 
the  direct  needs  of  the  warfighter  and  less  likely  to  focus  on  capabilities 
of  supporting  processes.  At  the  same  time,  a  significant  number  of  ICDs 
listed  net-centricity  and  C4/Cyber  as  supporting  FCBs. 

The  research  team  decided  early  on  to  capture  the  length  of  ICDs  as  the 
Capabilities-Based  Assessment  (CBA)  User’s  Guide  specifically  states  that 
ICDs  should  be  no  longer  than  10  pages,  with  separate  allowance  for  appen¬ 
dices  (JCS,  2009).  Figure  3  presents  the  average  ICD  page  length  without 
appendices;  quality  and  meticulousness  were  not  necessarily  correlated 
with  quantity  of  pages.  ICDs  were  meant  to  be  concise  documents  that  out¬ 
line  the  necessary  capabilities  while  still  answering  the  required  content. 

The  drastic  increase  in  length  of  ICDs  is  potentially  a  result  of  a 
change  in  the  process  by  which  capability  gaps  were  outlined.  As  with 
most  processes,  uncertainty  in  a  new  method  allows  for  an  increase  in 
the  breadth  and  depth  of  the  information  found  within  ICDs.  As  page 
length  has  been  steadily  decreasing  over  the  last  few  years,  it  would  sug¬ 
gest  that  sponsors  have  become  more  comfortable  with  the  process  and 
have  become  more  efficient  at  outlining  the  information  needed. 

One  final  note  concerning  page  length  was  to  evaluate  the  relation  of 
page  length  to  Acquisition  Category  (ACAT)  level.  Would  larger  projects 
lend  themselves  to  taking  more  pages  to  explain  the  research  and  identify 
the  gaps?  These  two  factors  were  examined,  and  between  ACAT  Levels  I,  II, 
and  III  the  mean  page  length  was  25.53, 23.35,  and  21.02  respectively.  While 
the  difference  between  ACATs  I  and  III  are  statistically  significant  using  a 
t-test  with  an  alpha  of  .05,  the  difference  (on  average)  is  roughly  four  pages. 

Within  the  time  period  analyzed,  a  total  of  2,779  gaps  were  identi¬ 
fied;  the  average  number  of  gaps  identified  in  an  ICD  are  shown  in  Figure 
4.  Additionally,  Figure  4  illustrates  the  fiuctuation  in  the  number  of 
ICDs  validated  each  year.  The  GAO  (2008)  report  noted  that  JCIDS  was 
ineffective  in  properly  prioritizing  capabilities  and  suggested  that  nearly 
all  ICDs  submitted  were  accepted.  Since  the  inception  of  the  JCIDS  pro¬ 
cess,  2012  was  the  first  year  that  the  average  number  of  gaps  exceeded 
number  of  ICDs  validated.  This  suggests  that  ICDs  are  identifying  more 
gaps  per  document,  creating  documents  that  are  tackling  larger  and 
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more  complex  problems  than  before.  It  appears  that  the  JCIDs  process 
has  matured,  and  the  process  has  become  more  efficient  as  a  result  of 
the  GAO  report. 

The  research  team  noted  that  many  ICDs  had  “too  few”  gaps  identified 
(only  one  or  two,  or  none  at  all)  leading  to  the  conclusion  that  the  method¬ 
ology  employed  was  not  optimal  as  there  are  probably  more  gaps  that  have 
yet  to  be  identified,  and  several  documents  identified  “too  many”  gaps.  It 
was  very  difficult  to  understand  and  prioritize  identified  gaps  when  too 
many  were  identified  (several  contained  over  50  gaps). 

Figure  5  is  a  representation  of  the  most  frequently  used  methodolo¬ 
gies  from  2002  to  2012,  displaying  the  percentage  of  ICDs  covered  by 
the  methodology.  The  top  five  methodologies  were  chosen  for  represen¬ 
tation  as  they  represented  those  methodologies  that  were  implemented 
in  greater  than  10  percent  of  ICDs,  whereas  the  remaining  methodolo¬ 
gies  were  typically  used  in  one  to  two  ICDs  only.  Each  ICD  employed 

FIGURE  4.  AVERAGE  NUMBER  OF  GAPS  IDENTIFIED  COMPARED 

TO  NUMBER  OF  VALIDATED  DOCUMENTS 
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several  methodologies  so  the  percentages  will  not  sum  to  100  percent. 
A  variety  of  analytical  techniques  maybe  appropriate  depending  on  the 
type  of  analysis  being  conducted.  As  an  example,  intelligence-based 
assessment  would  likely  be  an  appropriate  technique  for  identifying  a 
strategic  capability  gap  requiring  a  new  weapon  system,  but  not  appro¬ 
priate  for  identifying  the  need  for  a  new  inventory  system  for  the  Defense 
Commissary  Agency. 

Doctrine,  Organization,  Training,  Materiei,  Leadership, 
Personnei,  Faciiities-Poiicy 

The  research  team  observed  at  least  two  interpretations  of  the 
Doctrine,  Organization,  Training,  Materiel,  Leadership,  Personnel, 
Facilities-Policy  (DOTMLPF-P)  analysis  within  the  ICDs.  The  analy¬ 
sis  sometimes  took  the  course  where  ICDs  identified  DOTMLPF-P 
categories  of  nonmateriel  solutions  that  could  satisfy  capability  gaps, 
while  others  took  the  second  interpretation  where  ICDs  considered 
the  DOTMLPF-P  implications  of  their  proposed  materiel  solution. 
Defense  Acquisition  University  training  for  DOTMLPF-P  distinguishes 
between  these  uses  and  indicates  that  the  ICD  should  focus  on  the  for¬ 
mer  approach  as  the  latter  is  addressed  in  later  stages  of  the  acquisition 
process  (Defense  Acquisition  University,  2014). 


FIGURE  5.  TOP  METHODOLOGIES  USED 


Leading  Methodologies 


Matrix  Alternatives  Planning  traceability  Experts 

Matrix 
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We  also  observed  a  wide  range  of  quality  in  these  analyses.  Many 
ICDs  contained  rote  statements  declaring  the  insufficiency  of  these  non¬ 
materiel  approaches  to  close  capability  gaps.  To  paraphrase  an  example, 
several  ICDs  stated  that  “DOTMLPF  solutions  were  considered...,  but 
adjustments  or  improvements  in  these  areas  will  have  minimal  impact 
to  mission  satisfaction.”  Though  not  every  capability  gap  can  be  met  with 
nonmateriel  solutions,  such  “box  check”  DOTMLPF-P  analyses  offer  no 
value  to  the  requirements  validation  process. 

In  contrast,  several  analyses  reflected  a  concerted  effort  to  find 
nonmateriel  solutions  to  supplement  the  proposed  materiel  solution. 
One  example  of  this  level  of  analysis  is  the  Air  Force’s  Advanced  Pilot 
Training  ICD.  In  its  DOTMLPF-P  analysis,  the  Service  employed  a  three- 
phase  process;  first,  brainstorming  and  combining  possible  solutions; 
second,  conducting  quantitative  analysis  on  a  subset  of  the  best  of  the 
proposed  solutions;  and  third,  conducting  a  qualitative  assessment  of  the 
final  list  of  proposed  solutions.  Not  all  of  the  nonmateriel  solutions  were 
deemed  feasible  or  prudent,  but  several  were  included  as  part  of  the  final 
recommendations.  Further  explanations  of  how  the  Air  Force  conducted 
this  analysis  are  found  in  the  ICD  and  its  attachments  on  KM/DS. 

Recommendations  and  Guidelines 

Through  the  analysis  the  team  observed  a  variety  of  interpretations 
of  howto  write  an  ICD.  In  general,  analytical  rigor  could  be  stronger. 
In  a  fiscally  constrained  environment,  the  importance  of  documenting 
analysis  is  magnified,  and  many  ICDs  fell  short  of  careful  documentation 
of  analysis.  Another  observation  is  that  most  of  the  ICDs  were  submitted 
by  the  Services  and  very  few  by  Joint  sponsors.  This  is  not  surprising  as 
individual  Services  organize,  train,  and  equip  their  forces;  it  is  expected 
that  capability  gaps  will  continue  to  be  identified  by  the  Services. 

Useful  Analytical  Techniques 

Several  ICDs  utilized  subject  matter  experts  (SMEs)  to  identify 
capability  gaps  and  recommend  solutions.  One  way  to  incorporate  SME 
input  into  a  more  rigorous  fashion  is  by  employing  the  Delphi  Technique. 
In  this  method,  the  researcher  works  with  10-15  experts  to  identify, 
further  define,  and  determine  the  importance  of  an  issue  in  their  area 
of  expertise  (Tinstone  &  Turoff,  1975).  Using  the  Delphi  method  when 
SMEs  are  available  is  one  way  to  add  analytical  rigor  to  the  ICD  process. 
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Though  not  possible  for  all  ICDs,  several  documents  included  a  life- 
cycle  cost  summary  that  was  effective  in  communicating  the  costs  of 
the  capability  gap.  If  the  proposed  solution  is  expected  to  reduce  some 
recurring  cost,  presenting  those  numbers  can  make  a  convincing  case 
to  the  reader. 

In  the  Appendix  to  this  article,  the  authors  provide  a  list  of  addi¬ 
tional  analytical  techniques  along  with  a  short  description  of  each.  This 
resource  is  intended  to  assist  ICD  writers  and  project  managers  in  select¬ 
ing  a  methodology  or  methodologies  appropriate  for  their  document  or 
project.  References  are  provided  to  direct  interested  readers  to  source 
documents  with  additional  descriptions  of  each  methodology. 


One  way  to  incorporate  SME  input  into  a  more 
rigorous  fashion  is  by  employing  the  Delphi 
Technique.  In  this  method,  the  researcher  works 
with  10-15  experts  to  identify,  further  define,  and 
determine  the  importance  of  an  issue  in  their  area 
of  expertise  (Linstone  &  Turoff,  1975). 


Architectural  Enhancements 

Nearly  all  existing  ICDs  present  a  High-Level  Operational  Concept 
Graphic  (OV-1)  depicting  the  proposed  solution(s).  A  previous  Air 
Force  Institute  of  Technology  researcher  identified  several  additional 
Department  of  Defense  Architecture  Framework  (DoDAF)  prod¬ 
ucts  that  could  be  useful  to  present  within  the  ICD  (Hughes,  2010). 
The  Capability  Taxonomy  (CV-2),  Capability  Dependencies  (CV-4), 
Capability  to  Operational  Activities  Mapping  (CV-6),  as  well  as  the 
Operational  Resource  Flow  Description  (OV-2)  and  Operational  Activity 
Decomposition  Tree  (OV-5a)  are  products  now  required  by  JCIDS  for 
the  ICD. 

Hughes  also  found  value  in  including  the  Operational  Activity  Model 
(OV-5b)  and  Operational  Activity  to  Systems  Function  (SV-5).  The  OV-5b 
presents  capabilities  and  activities  and  their  relationship  among  activities. 
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inputs,  and  outputs.  The  SV-5  maps  systems  back  to  capabilities  or  opera¬ 
tional  activities.  Neither  is  currently  recommended  in  the  JCIDS Manual, 
but  could  be  presented  there  as  optional  architecture  products. 

Characteristics  of  Modei  iCDs 

Based  upon  analysis  of  the  data  that  were  examined  during  the  study, 
several  guidelines  or  best  practices  emerged.  The  best  written  ICDs 
provided  detailed,  but  relevant  analysis  without  being  too  wordy.  Here, 
we  propose  the  contents  of  a  model  ICD. 

The  most  fundamental  building  block  of  an  ICD  is  conformance  to 
JCIDS  standards  of  format  and  content.  The  JCIDS  Manual  presents 
a  logical  flow  of  the  document  from  gap  identiflcation  to  flnal  recom¬ 
mendations.  The  Concept  of  Operations  should  illustrate  how  the 
described  capability  will  support  the  Joint  Force  Commander.  The 
JCAs  or  Universal  Joint  Task  List  pedigree  should  be  clear,  but  not  overly 
detailed.  Documents  that  rolled  up  capability  gaps  to  Tier  2  or  Level  2 
components  seemed  more  readable  than  those  that  traced  capabilities 
to  lower  levels.  A  document  that  acknowledges  extant  systems  is  more 
convincing  in  establishing  a  capability  gap. 

The  team  believes  that  a  concise  ICD  may  be  written  with  5-12  gaps 
identifled.  Page  lengths  may  vary  by  ACAT  level,  with  more  complex 
proposed  solutions  demanding  more  explanation,  but  the  ideal  ICD 
would  be  15-25  pages  in  length.  In  short,  a  well- written  ICD  will  follow 
the  prescribed  format,  clearly  deflne  its  necessity  to  the  Joint  mission, 
and  be  presented  in  a  clear  and  logical  manner.  Additionally,  the  ICD 
should  present  clear  MOEs  with  minimum  and  desired  values.  Good 
MOEs  allow  the  reader  or  evaluator  to  know  when  the  new  capability 
has  delivered  on  its  design  promises.  MOEs  are  sometimes  confused 
with  measures  of  performance  (MOPs).  Noel  Sproles  states,  “MOEs 
are  concerned  with  the  emergent  properties  or  outcomes  of  a  solution. 
They  take  an  external  view  of  a  solution  and  as  such  are  different  from 
MOPs,  which  are  concerned  with  the  internal  workings  of  a  solution” 
(Sproles,  2002). 

Table  2  compares  ICD  content  required  by  the  Capabilities-Based 
Assessment  (CBA)  User’s  Guide,  the  JCIDS  Manual,  and  recommenda¬ 
tions  based  on  our  analysis.  As  part  of  the  analysis,  the  team  identifled 
those  ICDs  that  implemented  and  followed  the  best  practices  identifled 
by  the  team.  These  ICDs,  shown  in  Table  3,  are  identifled  to  give  future 
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ICD  writers  and  functional  groups  examples  of  what  they  can  strive 
toward  to  make  clear  and  concise  documents  that  are  both  effective  and 
efficient. 


Future  Research  and  Conclusions 

Future  research  could  focus  on  the  relationship  between  the  ICD 
and  the  program  it  generates.  Can  the  utility  or  performance  of  a 
program  be  traced  to  the  description  of  the  initial  capability  gap  and 


TABLE  2.  COMPARISON  OF  CBA/ICD  CONTENT 


CBA  User's  Guide 

JCIDS  Manual 

Research  Team 

Purpose 

CONOPS/Desired 

Outcomes 

CONORS 

Background/Guidance 

Joint  Functional  Areas 

Relationship  to  Tier  2 
JCA/UJTL 

Objectives 

Description  of 

Required  Capability 
Gaps,  Overlaps, 
Redundancies 

Analysis  Techniques 
Used  with  Description 
of  Scope 

Scope 

Capability  Attributes/ 
Metrics 

Prioritized  List  of  5-12 

Capability  Gaps 

Methodology 

-Approaches 

-MOEs 

-Technological/Policy 

Opportunities 

Relevant  Threats/ 
Operational 

Environment 

Clearly  Defined  MOEs 
with  Threshold  and 

Objective  Values 

Organization/ 

Governance 

Proposals  for  Non¬ 
materiel  Solutions 

DOTMLPF-P  Analysis 

of  Nonmateriel 

Solutions 

Projected  Schedule 

Final 

Recommendations 

Clear  Final 

Recommendations 

Responsibilities 

Note.  CBA  =  Capabilities-Based  Assessment;  CONORS  =  Concept  of  Operations; 
DOTMLPF-P  =  Doctrine,  Organization,  Training,  Materiel,  Leadership,  Personnel, 
Facilities  -  Policy;  ICD  =  Initial  Capabilities  Document;  JCA  =  Joint  Capabilities 
Assessment;  JCIDS  =  Joint  Capabilities  Integration  and  Development  System;  MOEs  = 
Measures  of  Effectiveness;  UJTL  =  Universal  Joint  Task  List. 
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TABLE  3.  SAMPLE  OF  EXEMPLARY  ICDs 


Document  Name 
(Control  Number) 

Year 

Noteworthy  Items 

Data  Masked  (05- 
51947485-00) 

2005 

Layered  analytical  methods  resulted 

in  100  shortfalls  that  were  further 
clustered  and  examined-top  3 
presented  for  further  study 

Military  Operational 
Medicine  (07- 
65416952-00) 

2007 

Extensive  Doctrine,  Organization, 
Training,  Materiel,  Leadership, 

Personnel,  Facilities  (DOTMLPF);  lots 
of  prioritized  tables 

Aviation  Ground 

Support  (07- 
600735309-00) 

2007 

Prioritized  tables,  quantitative 
threshold  values,  good  DOTMLPF, 
multiple  methods  used  to  determine/ 
rank  nonmateriel  solutions 

Initial  Capabilities 
Document  for  Joint 

Improvised  Explosive 
Device  Defeat  (07- 
66686002-00) 

2007 

Performed  a  well-documented, 
thoughtful  DOTMLPF  analysis; 
references  three  assessments— Joint 
Staff  (J8),  Joint  Improvised  Explosive 
Device  Defeat  Task  Force  baseline,  and 
follow-on;  prioritized  tables 

Biometrics  in 

Support  of  Identity 
Management  (09- 
090146111-00) 

2008 

Detailed  analysis  including  Scenario- 
based  Planning  and  Risk  Analysis 

Advanced  Pilot 

Training  (10- 
99164267-00) 

2009 

Strong  DOTMLPF  analysis;  clear 
explanation  of  analytical  approach 
included  in  Appendices 

Vessel-to-Shore 
Bridging  (09- 
97169105-00) 

2009 

Gaps  have  numerous  subparts;  uses  a 
typical  but  good  example  of  capability 
prioritization/mapping  matrix 
(includes  Measures  of  Effectiveness 
[MOE]  and  Minimum  Values) 
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TABLE  3.  SAMPLE  OF  EXEMPLARY  ICDs  (CONTINUED) 


Document  Name 
(Control  Number) 

Year 

Noteworthy  Items 

Cross  Domain 

Enterprise  (10- 
112959174-00) 

2010 

Uses  a  typical  but  good  example  of 
capability  prioritization/mapping 
matrix  (includes  MOEs  and  Minimum 
Values);  recommends  mix  of  materiel 

and  nonmateriel  solutions 

Amphibious  Combat 
Vehicle  ICD  (11- 
151956055-00) 

2011 

Requirements  traceable  to  the  Joint 
Operating  Concept  vice  Universal  Joint 
Task  Lists;  uses  a  typical,  but  good 
example,  of  capability  prioritization/ 
mapping  matrix  (includes  MOEs  and 
minimum  values);  recommends  mix  of 

materiel  and  nonmateriel  solutions 

Personnel  Recovery 
(12-167465473-00) 

2012 

Succinct  document;  recommends 

materiel  and  nonmateriel  solutions 

Data  Masked  (12- 
159990107-00) 

2012 

Detailed  analysis  using  several 
techniques;  well-defined  MOEs 
including  Threshold  and  Objective 

Values 

requirement  definition?  Are  there  characteristics  of  an  ICD  that 
indicate  how  well  a  program  will  adhere  to  cost,  performance,  and 
schedule  expectations? 


Since  2002,  the  JCIDS  process  has  been  refined  and  enhanced.  There 
appears  to  be  a  convergence  in  the  formatting  and  content  of  many  ICD/ 
JCDs  since  2008.  While  the  quality  of  historical  ICDs  varies,  marked 
improvements  to  the  analysis  have  been  documented  since  2008,  pos¬ 
sibly  due  to  the  GAO  report  from  the  same  year. 

Through  research  of  the  current  methodologies  used  in  ICDs  since 
the  inception  of  the  process,  the  research  team  has  formulated  an  out¬ 
line  of  proposed  areas  upon  which  writers  and  implementers  can  focus. 
Future  writers  may  use  this  outline  as  well  as  a  series  of  DoD  guidelines 
to  provide  the  Joint  community  with  superior  ICDs  that  achieve  their 
goals  in  a  more  efficient  manner  with  minimal  processing  time. 
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APPENDIX 


Additional  Analytical  Techniques  to  Assist  Initial  Capabilities  Document  (ICD) 
Writers  and  Project  Managers 


Method 

Source(s) 

Explanation 

Usage  Context(s) 

Pre-Capabilities-Based  Assessment  (CBA) 

Scenario-based 

Planning 

Capabilities- 

Based 

Assessment 
(CBA)  User's 
Guide,  p.  87 
(Ringland  & 
Schwartz,  1998) 
(Hiam,  1990,  p. 
284) 

Technique  using  scenarios  to  define/give  structure 
to  an  otherwise  murky  strategic  future.  A  type 
of  brainstorming,  which  may  use  nominal  group 
technique  or  another  group  problem-solving 
technique. 

•  Assumptions/drivers  of  change  (identify  key 
variables  and  historical  trends) 

•  Develop  framework  for  drivers 

•  Produce  initial  miniscenarios  (vary  the  type: 
surprise-free,  radical,  and  in-between) 

•  Reduce  to  2  or  3  scenarios 

•  Write  scenarios 

•  Identify  issues  arising  (sensitivity  analysis  with 
scenarios'  impact  on  key  variables) 

Mostly  pre-CBA;  used  to 
build  portfolios;  however, 
can  be  used  in  a  CBA  (e.g., 
to  analyze  threats,  etc.). 

Strengths, 
Weaknesses, 
Opportunities 
and  Threats 
(SWOT) 

Analysis 

(Helms  & 

Nixon,  2010) 

Analyzes  internal  (strengths/weaknesses)  and 
external  (opportunities/threats)  factors  to  help 
guide  corporate  strategy  development.  Useful 
in  a  group  strategy  setting,  using  nominal  group 
technique,  or  another  group  problem-solving 
technique  (like  a  Group  Decision  Support  System, 
or  GDSS).  See  also  Porter's  5  Forces  and  Barney's 
Resource-based  View  for  more  specific  analyses. 

Mostly  pre-CBA;  used  to 
build  portfolios;  however, 
can  be  used  in  a  CBA  (e.g., 
to  analyze  threats,  etc.). 
Generally  criticized  for  its 
lack  of  depth  and  rigor. 

Porter's  5  Forces 
Analysis 

(Porter,  2008) 

Builds  on  the  "threats/opportunities"  side  of 

SWOT  to  explain  how  market  structure,  defined 
by  five  market  forces  (threat  of  entrants, 
supplier  power,  buyer  power,  intensity  of  rivalry, 
threat  of  substitutes)  and  one  additional  force 
(complementors/government/public)  drive  the 
content  and  performance  of  firms. 

Mostly  pre-CBA;  used  to 
build  portfolios;  however, 
can  be  used  in  a  CBA 
(e.g.,  to  analyze  threats, 
etc.).  Generally  criticized 
for  focus  on  external 
environment,  vice  internal. 

Barney's 
Resource-based 
View  (RBV) 

(Barney,  1991) 

Builds  on  the  "strengths/weaknesses"  side  of 
SWOT  to  explain  how  a  firm's  internal  resources 
(value  [V],  rareness  [R],  nonsubstitutability  [NS], 
imperfect  imitability  [//]),  lead  to  sustainable 
competitive  advantage  (SCA).  SCA  =  V  +  R  +  NS 
+  II  Must  have  first  three  to  achieve  competitive 
advantage,  and  all  four  to  achieve  SCA. 

Mostly  pre-CBA;  used  to 
build  portfolios;  however, 
can  be  used  in  a  CBA 
(e.g.,  to  analyze  threats, 
etc.).  Generally  criticized 
for  focus  on  internal 
environment,  vice  external. 
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Method 

Source(s) 

Explanation 

Usage  Context(s) 

Pre-Capabilities-Based  Assessment  (CBA) 

The  Project 
Management 
Diamond 
Approach 

(Shenhar  & 
Dvir,  2007) 

Uses  four  quadrants  of  Technology,  Complexity, 
Novelty,  and  Pace  to  define  the  size,  scope,  and 
risk  of  a  systems  engineering  product/project. 

Pre-CBA  (used  to  define  a 
product  portfolio),  CBA/ 
ICD  (developing  Measures 
of  Effectiveness  (MOE), 
Capabilities  Development 
Document  (CDD)  (defining 
system  risk). 

Market 

Segmentation 

Grid 

GAO  Report 
No.  07-388, 

p.n 

A  grid  that  compares  four  markets  (current/new 
customers  in  existing  segments/customers  in  new 
segments/new  customer  wants  and  needs)  to  four 
offering  types  (current  business/enhancement  to 
current  business/new  business/new  to  industry) 
to  position  portfolio  projects  into  four  categories 
(strike  zone/traditional/pushing  the  envelope/ 
white  space  opportunity).  A  method  of  analyzing 
business  risk  that  encourages  businesses  to  find 
the  right  mixture  of  categories  of  projects.  Similar 
to  Risk/Rewards  Matrix. 

Mostly  pre-CBA;  used  to 
build  portfolios;  however, 
can  be  used  in  a  CBA. 

Risk-rewards 

Matrix 

GAO  Report  A  grid  that  plots  "risks"  vs.  "rewards"  of  projects. 

No.  07-388,  Similar  to  Market  Segmentation  Grid  in  that  it 

p.  16  encourages  businesses  to  find  the  right  mixture 

(Hiam,  1990,  p.  of  categories  of  projects.  The  same  tool  can  be 

377)  used  to  compare  effectiveness  to  cost  in  the  AoA 

"Alternatives  Comparison"  step  (particularly 
useful  in  showing  confidence  levels  and  threshold 
values).  The  "GE  matrix"  version  of  this  maps 
"business  strength"  (internal)  vs.  "industry 
attractiveness"  (external).  The  circles  may  be 
subdivided  into  market  share/total  market  pies  to 
enhance  analysis.  Augments  SWOT. 

Mostly  pre-CBA;  used 
to  build  portfolios; 
however,  can  be  used  in 
a  CBA.  Strength  is  that 
the  confidence  level  of 
estimates  is  captured  (by 
the  size  of  the  circles). 

Defense ARJ,  July  2014,  Vol.  21  No.  3 : 716-749 


738 


APublication  of  the  Defense  Acquisition  University 


http://www.dau.mil 


APPENDIX  (Continued) 


Method  Source(s)  Explanation 


Usage  Context(s) 


Nominal  Group  (Sink,  1983) 
Technique 


A  brainstorming  technique  that  mixes  individual 
and  group  activities  to  attempt  to  increase  the 
amount,  diversity,  and  quality  of  ideas  generated. 
Many  variations,  but  follows  the  basic  process 
below: 

•  Individual  Brainstorming 

•  Sharing  Ideas 

•  Group  Brainstorming  (divergent) 

•  Group  Discussion 

•  Group  Brainstorming  (convergent) 

•  Voting/ranking 


Pre-CBA  strategic 
planning,  CBA  (developing 
capabilities/MOEs), 
Analysis  of  Alternatives 
(AoA)/ICD/CDD 
(developing  attributes/Key 
Performance  Parameters 
[KPPs]).  Technique 
strong  in  generating  many 
diverse  ideas  without 
arriving  at  Groupthink. 
Other  group  problem¬ 
solving  techniques  may  be 
superior  (e.g.,  GDSS),  but 
at  an  increased  process 
cost. 


Delphi  Technique  (Goodman, 
1987) 


A  type  of  brainstorming  that  uses  experts  to  a) 
identify  issues  in  their  area  of  expertise,  b)  further 
define  issues  in  their  area  of  expertise,  and  c) 
identify  the  importance  of  issues  in  their  area  of 
expertise.  Generally  uses  3-9  experts,  and  begins 
with  Nominal  Group  Technique,  using  future  rounds 
to  refine/reduce/prioritize  issues. 


CBA  ICD  (capabilities, 
MOEs),  and  AoA/CDD 
(attributes,  KPPs).  An 
example  of  an  "expert" 
systems  analysis 
technique.  Careful  choice 
of  experts  is  essential. 


CBA/ICD 


Capabilities- 

Capabilities- 

1)  Describes  capabilities  required  to  perform  a 

CBA.  Results  in  an  ICD 

Based 

Based 

mission 

(which  not  only  documents 

Assessment 

Assessment 

2)  Identifies  gaps  in  capabilities  and  associated 

the  CBA,  but  acts  as  a 

(CBA) 

(CBA)  User's 
Guide 

operational  risks 

3)  Establishes  a  requirement  to  address  gaps 

decision  document). 

Initial  Capabilities- 

Capabilities  Based 

Document  (ICD)  Assessment 
(CBA)  User's 
Guide 


1)  Describes/summarizes  Concept  of  Operations 
(CONOPS)  (-1  page  explanation  of  CONOPS) 

2)  Describes  guidance  (see  Requirements 
Traceability  Matrix) 

3)  Describes  capabilities  required  (includes 
MOEs/threshold  values) 

4)  Describes  capability  gaps  (prioritized,  if 
possible) 

5)  Summarizes  relevant  threats/operational 
environment 

6)  Proposes  nonmateriel  and  materiel  solutions 
(see  Doctrine,  Organization,  Training,  Materiel, 
Leadership,  Personnel,  Facllities-Policy 
[DOTMLPF-P]  Analysis) 

7)  Final  recommendation  (normally,  but  not 
necessarily,  a  materiel  solution) 


CBA/ICD.  The  ICD  is 
a  decision  document 
to  further  explore  an 
enhanced  capability  (result 
of  a  CBA).  Cornerstone 
document  in  the  Joint 
Capabilities  Integration 
and  Development 
System  (JCIDS)  process. 
Listing  to  the  left  is  not 
comprehensive. 
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Method 

Source(s) 

Explanation 

Usage  Context(s) 

Requirements 

Traceability 

Matrix 

Air  Force 

Instruction 
(AFI)  10-601 

Also  known  as  "house  of  quality,"  traces 
system  attributes  to  operational/user/strategic 
requirements.  Multiple  levels. 

CBA/MOE  (developing 
capabllities/MOEs),  AoA/ 
ICD/CDD  (developing 
system  attrlbutes/KPPs). 

Paired 

Comparisons 

(Blanchard  & 
Fabrycky,  2010, 

p.182) 

To  build  a  rank-ordered  list,  each  of  the  options 
is  presented  to  the  decision  maker  two  at  a  time 
(instead  of  all  at  once).  For  N  criteria  to  be  ranked, 
N(N  -  l)/2  pairs  must  be  compared.  Assumes 
transitivity  of  preferences. 

CBA/MOE/ICD 
(development  of  criteria). 
Rank-ordering  importance 
of  design  parameters/ 
capabilities/gaps. 

Porter's  Value 
Chain  Analysis 

(Hiam,1990, 
p.  415) 

(Porter,  1980) 

1)  Select  unit  of  analysis,  both  for  your 
organization  and  for  competitors 

2)  Identify  primary  value-adding  activities  (direct/ 
indirect/quality  assurance) 

Inbound/outbound  logistics,  operations, 
marketing/sales,  service 

3)  Identify  support  activities  (direct/indirect/ 
quality  assurance) 

Procurement,  technical  development,  human 
resource  management,  firm  Infrastructure 

4)  Identify  linkages  between  value  chain  activities 

5)  Study  the  value  chain  to  identify  sources  of 
competitive  advantage 

Pre-CBA,  AoA.  Much  like 
a  DOTMLPF-P  Analysis, 
the  value  chain  requires  a 
gap  analysis,  but  not  just 
internal  (between  self  and 
competitors),  and  not  just 
in  isolation  (focus  is  on 
interactions). 

Systems 

Definition  Matrix 

(Sage  & 
Armstrong, 
2000,  p.  98) 

Applies  general  systems  theory  to  define  both  the 
SCOPE  (needs/objectives/criteria)  and  BOUNDS 
(parameters/variables/constraints)  of  a  system 
(e.g.,  capability,  MOEs,  attributes,  KPPs).  No 
real  analytic  technique  used  to  define,  although 
defining  the  SCOPE  and  BOUNDS  of  a  system  can 
use  many  of  the  methods  contained  herein. 

CBA/ICD  (capabilities, 
MOEs),  and  AoA/CDD 
(attributes,  KPPs).  See  also 
Work/Product  Breakdown 
Structure  (WBS/PBS)  for 
a  technique  to  develop  the 
initial  listing  of  attributes. 

Input-Output 

Matrix 

(Sage  & 
Armstrong, 

2000,  p.102) 

Applies  general  systems  theory  to  define  inputs 
(intended/unintended)  and  outputs  (desired/ 
undesired)  and  begin  a  more  sophisticated 
discussion  about  refining  a  system,  such  as: 
situation,  expertise,  risk,  spillover  effects, 
knowledge,  viewpoints,  experience,  kind  of  need, 
frequency,  urgency,  limits,  and  tolerances.  As 
shown  In  Sage,  uses  a  WB  structure,  but  could 
also  use  a  PB  structure. 

CBA/ICD  (capabilities, 
MOEs),  and  AoA/CDD 
(attributes,  KPPs). 
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Method 

Source(s) 

Explanation 

Usage  Context(s) 

Rapid 

Application 

Development 

(RAD) 

(Mackay, 

Came,  Beynon- 
Davies,  & 
Tudhope, 

2000) 

RAD  uses  short,  iterative  design  cycles  to  produce 
working  prototypes  and  systems.  A  mixture  of 
paper  prototypes  (e.g.,  the  different  Department  of 
Defense  Architecture  Framework  [DoDAF]  views, 
use  cases,  screen  shots),  code  stubs  1  menus,  and 
models  may  all  be  used.  Many  types,  including: 

•  Joint  Analysis  and  Design  (JAD):  y2-day 
sessions  placing  developers  and  users  together. 
Developers  use  the  rest  of  the  day  to  build 
prototypes.  Lasts  approximately  1  week. 

•  extreme:  exploration  consists  of  users  writing 
story  cards  (use  case),  which  developers  analyze 
and  give  estimates  to  complete.  Business 

then  prioritizes  the  cards  by  usefulness  and 
developers  prioritize  by  risk.  Best  mix  of  cards 
selected  to  implement. 

Usually  used  when 
implementation 
is  more  important 
than  documentation; 
however,  the  process 
of  idea  generation  and 
documentation  makes  this 
technique  ideal  for  pre- 
CBA  and  CBA  activities. 
Technique  may  also  be 
used  in  early  systems 
engineering  (SE)  to  help 
define  systems  (assumes 
that  many  users  do  not 
"know  what  they  want  until 
they  see  it"). 

Use-cases 

Capabilities- 

Based 

Assessment 
(CBA)  User's 
Guide,  p.  87 

A  use-case  may  be  as  broad  as  a  story  outlining 
how  a  system  would  be  used  in  an  ideal 
circumstance  (or  multiple  circumstances), 
or  might  be  as  specific  as  a  Unified  Modeling 
Language  (UML)-based  diagram  outlining  a 
specific  system  interaction  that  can  be  used  to 
generate  an  engineering  prototype.  Many  ICDs 
iterate  1-4  possible  "scenarios  for  use,"  with  the 
resulting  scenarios  resembling  SE  use-cases. 

Normally  post-CDD; 
however,  technique  useful 
in  early  SE.  See  Scenario- 
based  Planning  for  a  similar 
technique  applied  to  large- 
scale  planning. 

Intelligence- 

based 

Assessment 

Existing  ICDs 

Used  either  to  further  define  a  capability  gap, 
or  to  further  define  the  "threats/operational 
assessment"  category,  this  item  usually  lists 
the  threat  as  defined  by  current  intelligence 
assessments,  as  well  as  the  reference  for  the 
applicable  intelligence  assessment. 

Pre-CBA,  CBA,  ICD. 
Analysis  type  is  present  in 
Operations  Plan/Concept 
of  Operations  Plan 
(OPLAN/CONPLAN),  so 
it  helps  trace  operational 
requirements/gaps  to 
those  documents. 

Work/Product 
Breakdown 
Structure  (WBS/ 
PBS) 

(Turner  & 
Cochrane, 

1993) 

May  be  defined  from  top-down  (decomposition), 
or  bottom-up  (engineering).  Begin  with  major 
items,  and  continually  ask  "what  comes  next," 
or  "what  is  this  component/objective  made  of." 

Stop  when  either:  1)  you  know  how  to  measure 
(objectives)  or  2)  a  reasonable  amount  of  work 
(i.e.,  "work  package").  "Decomposition"  risk  is 
that  not  all  end  items  are  identified,  leading  to 
inaccurate  estimate.  Engineering  risk  is  in  omitting 
important  integration  items,  or  nonproduct-related 
tasks  (i.e..  Project  Management). 

CBA/MOE  (developing 
criteria),  AoA/ICD/ 

CDD  (system  definition). 
Used  to  decompose 
requirements  or  work 
hierarchically.  May  then 
be  used  for  the  basis 
of  defining/estimating 
work,  cost,  MOE,  or  other 
decision  objectives/criteria. 
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Method 

Source(s) 

Explanation 

Usage  Context(s) 

Measure  of 

Effectiveness 

(MOE) 

Definition 

(Sproles,  2002, 
p.255) 

•  Request  to  formulate  MOEs 

•  Determine  viewpoint 

•  Determine  mission 

•  Decide  on  Critical  Operational  Issues  (COI),  i.e., 
"tasks/categories" 

•  Draft  MOEs  (creative/testable/consistent  with 
library/statement) 

•  Evaluate/Revise/Agree  on  MOEs 

•  Apply  MOEs 

CBA/ICD.  MOEs  are 
normally  high-level,  and 
one  might  expect  10-20  of 
them  in  an  ICD,  whereas 
a  CDD  might  contain 
hundreds  of  KPPs.  Modern 
ICDs  will  usually  contain 
MOEs  as  well  as  threshold 
values. 

Requirements 
Correlation  Table 

Manual  for  the 
Operation  of  the 
JCIDS,  2012,  p. 
B-31;  AFIIO- 
601,  p.  37 

Summary  of  all  desired  capability  characteristics 
listed  as  threshold/objective  values,  mapped  to 
their  Joint  Capability  Area  (JCA).  Three  tables: 

Key  Performance  Parameters  (KPPs),  Key  System 
Attributes  (KSAs),  Attribute.  Each  table  has  a  brief 
explanation  of  derivation/justification  of  attributes 
listed. 

ICD/CDD.  Helps  decision 
makers  and  acquisition 
community  decide  on  most 
important  attributes,  and 
the  threshold  1  objective 
values  those  items  must 
exhibit.  Note  that  JCAs, 

•  KPP:  System  attributes  considered  most  critical 
or  essential  for  an  effective  military  capability. 
Failure  to  meet  KPP  threshold  may  result  in 
program  reevaluation/reassessment. 

•  KSA  Table  (AF-only):  Only  the  most  critical 
system  attributes  are  included  and  prioritized. 

•  Additional  Attribute:  Same  as  KSA,  but  contains 
additional  items. 


listed  in  the  Manual  for  the 
Operation  of  the  JCIDS,  p. 
B-B-1,  can  be  used  to  assist 
in  attribute  definition  as 
early  as  the  CBA  process, 
as  well  as  to  derive  KPPs 
from  JCAs. 


Capability  Gap  Existing  ICDs 
Matrix 


Perhaps  the  most  common  table  arising  since 
2008  in  ICDs,  this  table  lists  (in  the  following 
order):  Priority,  Capstone  Concept  for  Joint 
Operations  (CCJO)  Key  Characteristics,  Capability, 
JCAs,  Parameters/Measures  of  Effectiveness, 
and  Minimum  Value  (for  Parameters).  Answers 
many  key  questions,  and  may  be  combined 
with  a  capability  gap  matrix.  See  Requirements 
Correlation  Table  and  Capability  Gap  Pairwise 
Matrix. 


CBA,  ICD.  This  table 
combines  capabilities, 
MOEs,  and  minimum 
values.  It  does  not 
directly  address  capability 
gaps  (unless  gaps  are 
incorporated). 
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Method 

Source(s) 

Explanation 

Usage  Context(s) 

Capability  Gap 
Pairwise  Matrix 

Capabilities- 

Based 

Assessment 
(CBA)  User's 
Guide,  p.  89; 
Existing  ICDs 

A  method  of  prioritizing  capability  gaps  with 
respect  to  each  other  by  pairwise  comparison 
(using  correlation  matrix).  Each  capability  is 
listed  both  on  the  rows  and  the  columns,  and 
compared  to  others  (1.00  is  "the  same  as,"  while 
0.00-  .99  is  "less  than,"  and  1.01-  >  is  "greater 
than").  The  relative  weight  of  items  to  each 
other  is  multiplicative  (with  2.00  being  "twice 
as  important  as").  Scores  are  summed  across 
rows  (and  normed,  if  desired),  and  then  rank- 
ordered  based  on  the  scores,  with  a  higher  score 
being  more  important.  Note:  One  variation  uses 
"stoplight"  (i.e..  Red,  Yellow,  Green)  to  highlight 
the  degree  to  which  an  attribute  (column) 
represents  a  "gap"  with  current  key  LUTE,  JCA,  etc. 
tasks  (tuple). 

CBA.  Technique  also 
useful  to  rank-order  MOEs 
(ICD)  and/or  criteria 
(AoA/CDD).  See  Pairwise 
Comparison  for  a  similar 
technique  exploring  the 
same  questions  (uses 
transitivity  to  justify  using 
fewer  comparisons). 

Scores,  rankings,  and 
"stoplight"  symbols  are 
qualitative  measures, 
assigned  at  the  discretion 
of  the  ICD  team. 

DOTMLPF-P 

Analysis 

Capabilities- 

Based 

Assessment 
(CBA)  User's 
Guide;  Manual 
for  the  Operation 
oftheJCtDS 

Any  analysis  that  includes  the  following  factors 
(and  their  potential  interactions):  Doctrine, 
Organization,  Training,  Materiel,  Leadership 

Policy  and  Education,  Personnel,  Facilities,  and 
Policy.  Important  to  consider  in  all  phases  of 
early  systems  analysis,  including:  a)  Gap  analysis 
(CBA),  b)  nonmateriel  solution  (CBA— most 
typical  use),  c)  nonmateriel  enablers  to  materiel 
solution  (CBA  and/or  CDD). 

CBA/MOE  (developing 
capabilities/MOEs),  AoA/ 
ICD/CDD  (developing/ 
rating  system  Attributes/ 
KPPs).  See  DOTMLPF-P 
Matrix. 

DOTMLPF-P 

Matrix 

Existing  ICDs 

A  matrix  showing  capability  gaps  and/or 
objectives  down  tuples  and  Y/N/P  answers  to 
DOTMLPF-P  on  each  column,  with  a  "rationale/ 
comments"  column. 

Y  =  gap  may  be  resolved  without  materiel 
development 

N  =  no  solution  currently  exists 

P  =  partial  solution  exists 

CBA/ICD/AoA/CDD. 

This  version  of  the  matrix 
is  tailored  toward  gap 
analysis,  specifically. 

May  have  other  uses;  see 
DOTMLPF-P  Analysis. 

Cross-interaction 

Matrix 

(Sage  & 
Armstrong, 

2000,  p.  110) 

A  correlation  matrix  showing  the  interactions 
between  system  objectives  (as  shown,  uses 
ordinal  "+f  "0,"  and  "-"  to  show  interactions, 
but  could  also  use  scalar  Capability  Gap  Matrix 
Measures). 

CBA/ICD  (capabilities, 
MOEs),  and  AoA/CDD 
(attributes,  KPPs). 

Frequency/ 

Investment 

Matrix 

(Williamson, 

1979) 

Recurrent  or  occasional,  but  nonspecific  market 
transactions  are  best  handled  by  classical 
(market)  contracts.  The  tendency  toward 
recurrent  and  idiosyncratic  transactions  tends 
to  favor  unified  governance.  May  also  explain 
boundary  of  firm,  vertical  integration,  and 
departmentalization  (consideration  for  funding 

CBA  work  via  contracts). 

Used  to  determine  type 
of  contract  one  might 
use  to  purchase  different 
types  of  services  on  the 
market.  Uses  transaction 
costs  (immeasurable)  as 
theoretical  mechanism  to 
explain. 
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Method  Source(s)  Explanation 


Usage  Context(s) 


AoA/CDD 


Analysis  of 

Alternatives 

(AoA) 


AFI 10-601; 
Handbook, 
pp.  14,  31,  33, 
45,  46,  47 


The  AoA  is  a  process,  consisting  of  four  basic 
sections:  1)  Effectiveness  Analysis,  2)  Cost 
Analysis,  3)  Risk  Analysis,  and  4)  Alternative 
Comparison.  Each  of  these  four  items  uses 
techniques  such  as  Decision  Evaluation  Matrix  to 
evaluate  alternatives  based  on  MOEs.  MOEs  may 
be  mapped  to  their  overarching  tasks  or  desired 
outcomes. 

•  Effectiveness  Analysis:  1)  Select  Mission  Tasks 
(MT),  MOE,  and  MOPs,  2)  Select  threats/ 
scenarios,  3)  Describe  alternatives,  4)  Determine 
level  of  detail,  5)  Identify  suitable  analysis  tools/ 
data  sources  (consider  including  sensitivity 
analysis) 

•  Cost  Analysis:  1)  sunk,  2)  research  and 
development,  3)  investment,  4)  operating/ 
support,  5)  disposal,  6)  baseline  extension,  7) 
prefielding 

•  Risk  Analysis:  see  Risk  Analysis 

•  Alternative  Comparison:  see  Decision  Evaluation 
Matrix  and  Risk-Rewards  Matrix.  The  AoA 
Handbook  shows  a  Decision  Evaluation  Matrix  with 
additional  columns  (for  risk  and  cost). 


AoA/ICD  (developing 
and  applying  MOE 
to  capabilities),  CDD 
(developing  and  applying 
criteria  to  alternatives). 
The  items  used  to  BOUND 
the  AoA  are  same  items 
used  to  BOUND  the  ICD. 
AoA  Handbook  gives 
guidelines  for  performing 
the  steps,  overview 
of  analysis  tools,  and 
modeling  suggestions. 
Finally,  AoAs  need  not 
identify  a  single  solution 
(in  fact,  they  may  identify 
a  suite  of  solutions 
that  meets  certain 
requirements). 


Decision 

Evaluation 

Display 


(Blanchard  & 
Fabrycky,  2010, 
p.187) 


Graphical  representation  of:  1)  alternatives  (A,  B, 
C);  2)  equivalent  cost/profit;  3)  other  criteria  (X, 

Y,  Z).  Although  not  strictly  a  2-dimensional  view, 
the  x-axis  is  structured  according  to  increasing 
cost/profit  of  alternatives,  and  the  y-axis  is  scaled 
with  relative  (ordinal,  i.e.,  less  than,  equal  to, 
more  than)  achievement  by  alternatives  of  the 
criteria.  (Note:  Normally,  these  would  be  separate 
graphs  for  each  criteria,  but  they  are  stacked  on 
top  of  each  other  to  simplify  the  display,  with  no 
implication  of  relevance  of  the  different  position  of 
each  criterion  on  the  y-axis  [except  with  reference 
to  itself]). 


CBA/MOE  (developing 
criteria),  AoA/ICD/ 

CDD  (applying  criteria). 
Organizes  information  on 
alternatives  and  degree  of 
compliance  with  criteria 
(including  threshold 
values)  while  still  allowing 
for  decision-maker  insight, 
intuition,  and  judgment. 
Not  intended  to  be 
mathematically  applied. 
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Method  Source(s)  Explanation 


Usage  Context(s) 


Decision 

Evaluation 

Matrix 


Decision  Tree 


(Blanchard  & 
Fabrycky,  2010, 
p.189) 


A  matrix  with  alternatives  on  the  x-axis  (as  a 
tuple),  and  three  items  on  the  y: 

•  Header  #1:  a  future  not  under  the  control  of  the 
decision  maker  ("state  of  nature") 

•  Header  #2:  the  probability  (p)  of  that  future 

•  Each  cell:  evaluation  (E)  measure  (positive 
or  negative)  of  [alternative  x  future];  may  be 
subjective  (i.e.,  categorical)  or  objective  (e.g., 
monetary  values) 

Possible  decision-making  criterion  (to  select  most 
desirable  alternative): 

•  Aspiration  level:  setting  desired  min  and  max 
levels  for  each  criterion,  or  for  all  criteria  as  a 
whole 

•  Most  probable  future:  useful  if  one  probability 
dominates 

•  Expected  value  (EV):  EV  =  Z(E  X  p)  where  Ip  = 
1.00  useful  for  repetitive  environment 

-  Laplace:  if  p  unknown  for  each  alternative, 
divide  1.00  by  number  of  alternatives 

-  Maximin:  best  alternative  given  the  worst 
possible  outcome 

-  Maximax:  best  alternative  given  the  best 
possible  outcome 

-  Minimax  (includes  "regret"):  best  outcome 

-  outcome  for  "a,''/"s-";  attempts  to  calculate 
opportunity  cost  of  a  decision 

-  Hurwicz  rule:  assigns  an  optimism  index  from 
0-1.0  (assumes  linearity) 


CBA/MOE  (developing 
criteria).  AoA/  CD/ 

DD  (applying  criteria). 
Considers  alternatives/ 
criterion  of  effectiveness 
in  past/present,  but  also 
alternatives/possible 
future  conditions  [of 
use].  Assumes:  all  viable 
alternatives  considered, 
all  possible  futures 
identified,  all  futures 
and  [alternatives  x 
futures]  are  orthogonal, 
occurrence  of  specific 
future  is  unknown 
(otherwise,  matrix 
simplifies  to  a  vector  of 
evaluation  measures). 
Limitation:  each  of  these 
methods  yields  different 
results. 


(Kirkwood, 

2002) 


Calculates  an  expected  value  (EV)  for  each  of 
a  number  of  possible  options,  exploring  what 
happens  if  selection  leads  to  success  or  failure. 
May  include  a  "none  of  the  above"  option.  One 
common  use  is  to  include,  add  together  the  cost  of 
each  of  the  options  with  their  expected  payout  to 
generate  the  evaluation  (E)  measure. 

•  EV  =  Z(E  X  p),  where  Zp  =  1.00  for  the  outcome 
of  each  decision.  Most  useful  for  repetitive 
environment;  otherwise,  the  EV  metric  has  no 
inherent  meaning  (although  often  shown  as 
monetary  value,  $). 


C  BA/C  D/Ao  A/C  DD. 
Amenable  to  monetary 
decisions  that  can 
be  stated  in  Boolean 
(success/failure) 
terms.  Options  must  be 
orthogonal.  May  also  be 
used  to  model  multiple, 
sequential  decisions. 
See  Decision  Evaluation 
Matrix  for  an  additional 
application  of  this 
technique. 
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Method 

Source(s) 

Explanation 

Usage  Context(s) 

Optimization 

(Blanchard  & 

E  =  f(X,  Y,  y.) 

CBA/MOE  (developing 

Modeling/Linear 

Programming 

Fabrycky,  2010, 
p.177) 

X  =  Design  variables  (factors  that  define  design 
optimization  space) 

criteria),  AoA/ICD/ 

CDD  (applying  criteria). 
Determining  effectiveness 

Yj  =  Design-dependent  variables  (under  control  of 

of  a  system  based  on  a 

designers) 

model  of  that  system 

y  =  Design-independent  variables  (not  under 
designer-control) 

including  the  most  relevant 
variables.  Models  lack  of 
certainty  due  to  factors  not 
under  designers'  control. 

See  Decision  Evaluation 

Matrix  for  an  additional 
application  of  this 
technique. 

"Scorecard" 

Matrix 


(Sage  & 
Armstrong, 
2000,  p.  Ill) 


Yet  another  technique  to  compare  alternatives  to 
criteria,  this  time  with  the  emphasis  on  technology 
maturity  alternatives  (see  Market  Segmentation 
Grid)  crossed  with  the  "-ilities"— although  any 
combination  thereof  with  other  techniques  in  this 
listing  could  be  used. 


CBA/ICD  (capabilities, 
MOEs),  and  AoA/CDD 
(attributes,  KPPs). 


Utility 

(Indifference) 

Curves 


(Brosh,  1985, 
p.70) 


Having  developed  a  decision  tree  with  monetary 
outcomes  (but  not  yet  assigned  probabilities  of 
outcomes),  it  is  possible  to  query  the  decision 
maker  as  to  the  amount  deemed  acceptable 
as  a  guaranteed  payout  instead  of  accepting 
the  probabilities  of  payouts  represented  in  the 
decision  tree.  Varying  the  probabilities  and  re¬ 
asking  this  question  allows  one  to  create  a  utility 
curve,  with  the  payout  on  the  x-axis  and  utilities 
on  the  y-axis.  The  "risk-neutral"  decision  maker's 
utility  curve  is  negative  first  derivative  (positive, 
but  decreasing),  while  the  "risk-averse"  is  a 
positive  first  derivative  (positive,  but  increasing). 


Answers  question  of 
decision  maker's  risk- 
averse/neutral/seeking 
nature,  i.e.,  is  valuation  of 
marginal  utility  of  money 
decreasing/constant/ 
increasing?  Determines 
whether  to  use  minimin, 
minimax,  maximax. 

Paired  with  Decision 
Evaluation  Matrix  to  model 
alternative  preference 
in  terms  of  "utility"  vice 
"monetary." 


Weighting 


(Blanchard  &  Weights  (W)  must  sum  to  1.00  (100%)  for  each 
Fabrycky,  2010,  criterion.  Ratings  (R)  based  on  whatever  sca/ar 
p.  185)  rating  schema  one  devises  (does  not  work  for 

ordinal/categorical  ratings). 

Weighted  Rating  =  W  xR 

•  Tabular  display:  results  indicate  how  close  each 
alternative  comes  to  the  ideal. 

•  Graphical  additive:  results  indicate  the  overall 
contribution  of  the  rating  in  each  category  to  the 
overall  desirability  of  the  alternative. 


CBA/MOE,  AoA/ICD/ 
CDD.  Choosing  across 
a  number  of  design 
alternatives  when 
categories  are  not  of  equal 
importance  (see  systematic 
elimination  for  similar 
method).  Caution  is  advised 
in  developing  both  criterion 
and  weighting,  as  well  as  in 
interpreting  two  alternatives 
that  end  up  rating  near  each 
other  on  the  scale. 
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Method  Source(s)  Explanation 


Usage  Context(s) 


Z-score 

Transformation 


(Daszykowski, 
Kaczmarek, 
Vander  Heyden, 
&  Waiczak, 
2007) 


For  items  collected  using  ratio/continuous  data  for  AoA. 
which  an  expected  value  (mean)  and  dispersion 
(standard  deviation)  are  known,  application  of  a 
z-transformation  can  re-score  an  item  (results  in 
a  number  between  -1.0  and  -rl.O).  Items  can  then 
be  further  transformed  by  weighting  or  another 
technique  and  be  comparable  across  different  items 
(e.g.,  "time-to-implement"  vs.  "distance"). 


National  Family  (Hiam,  1990,  p. 
Opinion  (NFO)  273) 

Product  Analysis 


1)  Survey  customer  attitudes  to  obtain  rankings  of 
importance  of  product  attributes  and  a  rating  of 
the  overall  product  (Likert-type  scale:  5-point, 

-2  to  +2).  Likert  scales  are  commonly  used  in 
surveys  to  measure  attitudes. 

For  example: 

My  current  level  of  job  satisfaction  is: 

1  2  3  4  5  6  7 

Extremely  Extremely 

Unsatisfied  Satisfied 


AoA.  A  method  like  this 
compares  the  perceived 
"gap  severity"  with 
the  "importance"  of 
an  attribute  in  order  to 
assist  the  researcher  in 
prioritizing  the  attributes. 


or  in  the  case  of  the  NFO  Product  Analysis: 

Flow  well  does  the  product  meet  the  desired 
attribute  (x)? 

-2  -1  0  +1  +2 

Not  nearly  Far  too 

enough  of  X  much  of  x 


2)  Use  stepwise  linear  regression  to  determine  most 
important  attributes  to  overall  ratings  (calculate 

then  "Importance  Index":  R^|„j/  R\,). 

3)  Graph  the  "importance  index"  vs.  mean  ratings 
for  each  attribute.  Items  on  upper  corners  are 
those  worth  investing  effort  into. 


Systematic 

Elimination 


(Blanchard  & 
Fabrycky,  2010, 
p.183) 


Do  not  consider  weights,  nor  trade-offs  across 

alternatives.  May  use  scalar  or  categorical  ratings. 

•  Compare  alternatives  against  each  other  (norm- 
referencing;  will  establish  dominance  between  two 
options  [drop  the  lower  one]). 

•  Compare  alternatives  against  a  standard  (criterion- 
referencing:  1)  retaining  if  meets  standard  for  at  least 
one  criterion,  or  2)  retaining  if  meets  standard  for  all 
criterion). 

•  Comparing  criteria  across  alternatives  (after  ranking 
criterion:  1)  choose  best  alternative,  break  ties  with 
the  second  most  important  criterion,  or  2)  examine 
one  criterion  at  a  time,  comparing  the  alternatives 
and  eliminate  those  not  meeting  minimum 
standard). 


CBA/MOE  (developing 
criteria),  AoA/ICD/ 

CDD  (applying  criteria). 
Choosing  across  a  number 
of  design  alternatives. 
Outcomes  can  be 
specified  for  all  criteria 
and  alternatives.  May 
use  to  select  best  option, 
or  to  determine  which 
of  a  number  of  options 
meet  minimum  criteria 
for  further  inclusion.  See 
weighting  for  another 
similar  method. 
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Method 

Source(s) 

Explanation 

Usage  Context(s) 

Sahid's 

Consequences 

Table 

(Hammond, 
Keeney,  & 

Raiffa,  1998) 

Lists  alternatives  across  the  columns  and  key 
attributes/decision  criteria  down  tuples.  The 
goal  of  this  table  is  not  to  combine  disparate 
data  types,  but  rather  to  search  for  options  that 
clearly  "dominate"  other  options.  The  "dominated" 
options  are  then  eliminated  systematically. 

AoA.  Because  it  is  an 
initial  screening  process,  it 
reduces  options/simplifies 
choice;  however,  ensure  the 
most  important  attributes 
are  screened  first. 

Even  Swaps 

(Hammond, 
Keeney,  & 

Raiffa,  1998) 

A  more  sophisticated  analysis  using  Sahid's 
Consequences  Table,  "how  much  of  one  attribute 
are  you  willing  to  swap  for  an  increase/decrease 
in  the  other?"  In  this  way,  attributes  of  key  interest 
can  be  made  comparable  by  trading  up/down 
other  attributes.  This  is  one  form  of  sensitivity 
analysis. 

AoA.  Does  not  treat 
alternatives  as  exclusive; 
encourages  decision  maker 
to  look  for  (not  listed) 
alternatives  to  satisfy 
"swapped"  items. 

Risk  Analysis 

AoA  Handbook, 
p.  40 

Risks  are  categorized  by  Severity  (S,  i.e., 
consequence)  and  Probability  (p,  e.g.,  likelihood). 

If  each  risk  is  assigned  a  number  from  0.00  - 1.00 
for  both  categories,  then  a  composite  risk  index 
can  be  calculated  using:  CR  =  S  x p,  and  a  risk 
matrix  can  be  used  to  plot  the  results.  Risk  may 
then  be  avoided,  accepted,  transferred,  and/or 
mitigated.  Some  add  three  columns  to  a  risk  table 
to  add  how  the  risk  was  managed,  the  resultant 
risk,  and  any  secondary  risks  that  risk  mitigation 
created. 

AoA.  Technique  uses 
qualitative  assignment  of 
risk  values.  Normally,  risks 
are  assumed  orthogonal 
(however,  risk  interactions 
can  be  modeled  with  this 
technique). 

Sensitivity 

Analysis 

(Blanchard  & 
Fabrycky,  2010, 
pp.  589,  614) 

A  generic  category  of  tools  that  plots/graphs/ 
calculates  the  relationship  between  changing 
variables,  giving  an  idea  of  how  a  modification 
in  one  variable  affects  others.  Plotting  different 
alternatives  on  the  same  axis  gives  an  idea  of 
the  favorability  of  one  option  versus  the  other  in 
the  trade  space  measured  (a.k.a.  the  "breakeven 
point").  Examples:  Pareto  chart  (a  line  or  bar 
graph  displaying  results  ordered  by  frequency  of 
occurrence),  scatter  plot,  cost/year  plot. 

Primarily  AoA/CDD,  but 
can  be  used  in  CBA/ICD. 

Cost  Breakdown 
Structure  (CBS) 

(Blanchard  & 
Fabrycky,  2010, 
p.  577) 

Similar  to  a  WBS/PBS,  a  CBS  breaks  all  costs  down, 
either  by  product,  cost  center,  or  development 
phase.  Blanchard  and  Fabrycky  call  this  a  "functional" 
breakdown).  A  typical  CBS  might  include  items  such 
as:  research  and  development  cost,  production/ 
construction  cost,  operations  and  maintenance 
cost,  retirement  and  disposal  cost.  Many  of  the  cost 
categories  included  in  a  CBS  are  standardized  items 
in  the  finance  community,  and  each  has  estimation 
technique(s)  associated  with  it.  Costs  are  often 
captured  on  a  Cost  Coilection  Worksheet. 

AoA/CDD.  The  U.S. 
military  does  not  normally 
perform  some  of  the  key 
items  included  in  a  CBS; 
therefore,  estimates  in 
these  areas  may  not  be 
reliable  (or  else  the  military 
might  contract  the  cost 
estimate). 
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Method 

Source(s) 

Explanation 

Usage  Context(s) 

Cost  Collection 
Worksheet 

(Blanchard  & 
Fabrycky,  2010, 
p.586) 

Basic  mechanism  used  to  gather  and  report 
costs  generated  by  a  CBS.  Much  like  a  WBS, 
costs  are  broken  down  by  function  and 
subfunction  (and  the  associated  cost  categories) 
in  the  tuples,  while  the  cost  by  program  year, 
total  (actual),  total  net  present  value  and  % 
contribution  are  in  columns. 

AoA/CDD.  Compares 
programs  by  cost  center/ 
year,  or  cost  profile  (since 
profile  by  center/year  is 
accessible  to  viewing). 

Parameter-based 

Costing 

(Blanchard  & 
Fabrycky,  2010, 
p.581) 

One  of  the  four  types  of  cost  estimating, 
parametric  analysis,  involves  determining  key 
parameters  that  drive  cost  (historically),  then 
using  these  parameters  to  estimate  future  costs. 

AoA/CDD.  Only  as  good 
as  past  information  and 
current  judgment. 

Activity-based 

Costing 

(Blanchard  & 
Fabrycky,  2010, 
p.581) 

A  method  directed  toward  "detailing  and 
assignment  of  all  costs  to  the  activities  that 
cause  them  to  occur,"  in  an  effort  to  include 
traceability  (for  items  historically  difficult  to 
track;  i.e.,  indirect  costs  like  "overhead"). 

AoA/CDD.  May  be  at  odds 
with  WBS/PBS  methods 
of  tracking  costs  (because 
functions  like  project 
management  spread  across 
multiple  cost  centers). 

Life-cycle  Cost 
Summary 

AoA  Handbook, 
p.37 

Breaks  out  life-cycle  costs  two  ways:  1)  by 
alternative  and  life-cycle  phase,  2)  by  budget 
category  and  life-cycle  year  (any  combination  of 
these  is  acceptable,  based  on  the  requirement). 

AoA. 

Money  Flow 
Modeling 

(Blanchard  & 
Fabrycky,  2010, 
p.176) 

Considers  present  equivalent  (PE),  annual 
equivalent  (AE),  or  future  equivalent  (EE) 
amount,  as  well  as  internal  rate  of  return  and 
payback  period. 

PE,  AE,  or  FE  =  f(F,,  i,  n) 

t  =  0,1,2, ... ,  n  (salvage  value/cost  added  at  end 
of  final  year) 

ICD/AoA/CDD  (economic 
AoAs).  Calculating  outlay 
and  payback  of  a  system 
over  its  acquisition  and 
utilization.  See  Decision 
Evaluation  Matrix  for  an 
additional  application  of 
this  technique. 

=  positive  or  negative  money  flow  at  end  of  year  t 

1  =  annual  interest  rate 

n  =  number  of  years 
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